Evaluating child survival programmes.

a Department of Social Medicine, Federal University of Pelotas, Pelotas, Brazil. b Bloomberg School of Public Health, Johns Hopkins University, Baltimore, MD, United States of America. Correspondence to Cesar G Victora (e-mail: [email protected]). Renewed global interest in newborn and child survival has generated many programmes in lowand middle-income countries. There is an urgent need for a common evaluation framework to allow comparison of health outcomes and costs within and across these countries.1 The current generation of child survival programmes aims to deliver defined sets of biological and behavioural interventions that have proven effective in smaller-scale trials. Unlike efficacy studies, effectiveness evaluations assess whether interventions result in expected health improvements when scaled up under routine conditions. Also, in contrast to efficacy studies, political considerations often prevent investigators from controlling where these programmes are implemented, so that non-randomized designs are often the only option. A major change in approach2 is needed if future evaluations are to meet the needs of national governments and donors for rigorous assessments of the results of their investments both in child survival, in particular, and in health in general. Current evaluations assume that programmes are implemented only in a few “intervention” districts and not in the rest of the country. Changes in service provision, utilization, coverage and sometimes health impact are documented over time, and gains in the intervention districts relative to comparison areas are usually attributed to the programme, i.e. any improvement is a direct result of programme inputs. The underlying counterfactual is that, in the absence of the programme, outcomes in the target districts would have shown trends similar to those in the comparison areas. Several types of bias may compromise this comparison, including selection biases (districts chosen for the programme may have special characteristics that favour implementation) and confounding factors. Despite its limitations, this traditional evaluation design remains popular because there are no feasible alternatives; if properly conducted and analysed, these evaluations often provide valuable information. Our recent experience in evaluating large-scale child health initiatives suggests that this traditional design seldom allows valid attribution in the current development context. “Untouched” comparison areas are rarely available because multiple child survival interventions are being scaled up with greater or lesser success in different areas. As a consequence, populations in the comparison area are often exposed to biological and behavioural interventions similar to those in the programme area. For example, our recent evaluation of the Accelerating Child Survival and Development initiative in Mali (supported by the United Nations Children’s Fund) demonstrated that several of the comparison districts were receiving virtually the same interventions from other donors as the programme area. To further complicate the issue, child survival is influenced by myriad factors other than by individual programmes. At least four categories of contextual factors must be considered in attributing reductions in child mortality:3 (i) preexisting health services, whether public or private; (ii) new health programmes other than the one being evaluated; (iii) interventions in other sectors that may affect health (e.g. water, sanitation or education); and (iv) overall socioeconomic and environmental conditions. The broad framework of determinants of child health requires that evaluations extend beyond the health sector. Partners should work with national and local governments to support nationwide assessments covering multiple programmes. These assessments can be designed to meet the evaluation needs of governments and their partners in maternal, newborn and child health. They can also serve as a basis for evaluating other health programmes such as reproductive health and disease control, including those for malaria and HIV/AIDS, and efforts aimed at strengthening health systems.4 We propose three first steps when building an evaluation platform: 1) Develop and regularly update a district database that includes demographic, epidemiological, socioeconomic and health infrastructure variables, derived from sources such as censuses, economic surveys, poverty maps, service availability censuses, etc. 2) Conduct an initial survey, to be repeated every three years or so, to measure coverage levels for proven interventions and health status; ideally this survey would allow estimation of mortality and prevalence of biomarkers, being representative at district or – at the very least – provincial level. 3) Establish a continuous monitoring system for documenting provision, utilization and, ideally, quality of interventions at district and, if possible, sub-district levels, with mechanisms for prompt reporting to local, national and international audiences. This platform would allow multiple analyses based on ecological designs, with the units of intervention being the district or province. It would support the comparison of various combinations of interventions and delivery strategies – including the assessment of a range of intervention intensities – in regard to changes in health impact measures, while considering confounding factors. This approach is likely to cost less than the aggregate costs of conducting multiple separate evaluations and to generate more and better information about the effects of specific programmes, alone and in combination. The Paris Declaration on Aid Effectiveness called for independent cross-country monitoring and evaluation of development efforts.5 Will national governments, international partners and research funders stand up to the challenge? ■